fix(cli): expose sim screenshot orientation metadata#29
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This pull request adds machine-readable simulator screenshot orientation metadata to the triton sim screenshot --json host-action envelope (to avoid agents assuming display-normalized orientation), and introduces a shorter iOS deep-link smoke entrypoint (triton app go <url>) that defaults to the “wait-ready + snapshot + JSON” behavior.
Changes:
- Extend
HostActionOutputwith optionalscreenshotmetadata and add a schema output contract forhost.simulator-screenshot-metadata(including PNG IHDRpixelWidth/pixelHeight). - Add
triton app go <url>and update runtime transport “next actions” / workflow plan commands to preferapp goover longopen-url --wait-ready --snapshot --json. - Add/adjust tests and documentation to codify the raw framebuffer orientation semantics and the new command surface.
Reviewed changes
Copilot reviewed 16 out of 16 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| Sources/TritonKitCLI/CLISchemaHostCommands.swift | Updates schema shapes/semantics and registers the new screenshot metadata contract; adds app go subcommand to schema and updates nextCommands. |
| Sources/TritonKitCLI/CLISchemaContracts.swift | Adds screenshot field to host-action contract and introduces host.simulator-screenshot-metadata output contract. |
| Sources/TritonKitCLI/CLIRuntimeTransport.swift | Switches capability next-actions and workflow plan steps to use triton app go instead of long open-url invocations. |
| Sources/TritonKitCLI/CLIHostRuntime.swift | Populates HostActionOutput.screenshot for sim.screenshot and implements PNG IHDR dimension parsing. |
| Sources/TritonKitCLI/CLIHostModels.swift | Adds HostSimulatorScreenshotMetadata model and threads it into HostActionOutput. |
| Sources/TritonKitCLI/CLIHostCommands.swift | Adds HostAppGo command and default iOS host-device selection fallback logic. |
| README.md | Documents triton app go and clarifies semantics vs app open-url. |
| docs-linhay/spaces/20260604-issue-26-sim-screenshot-orientation/README.md | Adds BDD-style documentation for Issue #26 orientation semantics/acceptance. |
| docs-linhay/spaces/20260604-issue-26-sim-screenshot-orientation/plans/implementation.md | Records implementation plan and validation commands for Issue #26. |
| docs-linhay/spaces/20260527-command-surface-optimization/plans/checkpoints/20260602-round-180-short-app-go-deeplink-entry.md | Adds checkpoint doc for introducing app go. |
| docs-linhay/scripts/verify-simulator-gate.sh | Adds a test filter gate covering URL placeholder canonicalization. |
| docs-linhay/memory/2026-06-04.md | Adds memory note summarizing screenshot metadata semantics and validation caveats. |
| docs-linhay/memory/2026-06-02.md | Adds memory note summarizing app go and schema/plan updates. |
| docs-linhay/dev/ai-cli-readable-control.md | Updates agent-facing CLI guidance to recommend app go as the short smoke entry. |
| CLI/Tests/TritonKitCLITests/SchemaFactSourceTests.swift | Updates schema assertions for app go, machine-readable defaults, and adds URL placeholder canonicalization coverage. |
| CLI/Tests/TritonKitCLITests/DeviceCrossPlatformTests.swift | Adds tests for screenshot metadata semantics and app schema examples; includes a minimal PNG writer helper. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+1564
to
+1579
| private func readPNGDimensions(path: String) throws -> (width: Int, height: Int) { | ||
| let url = URL(fileURLWithPath: path) | ||
| let data = try Data(contentsOf: url, options: [.mappedIfSafe]) | ||
| guard data.count >= 24 else { | ||
| throw RuntimeError("Screenshot metadata could not be read: PNG file is too short.") | ||
| } | ||
| let signature: [UInt8] = [0x89, 0x50, 0x4E, 0x47, 0x0D, 0x0A, 0x1A, 0x0A] | ||
| guard Array(data.prefix(8)) == signature else { | ||
| throw RuntimeError("Screenshot metadata could not be read: output is not a PNG file.") | ||
| } | ||
| guard String(data: data[12..<16], encoding: .ascii) == "IHDR" else { | ||
| throw RuntimeError("Screenshot metadata could not be read: PNG IHDR chunk is missing.") | ||
| } | ||
| let width = data[16..<20].reduce(UInt32(0)) { ($0 << 8) | UInt32($1) } | ||
| let height = data[20..<24].reduce(UInt32(0)) { ($0 << 8) | UInt32($1) } | ||
| return (Int(width), Int(height)) |
| ("stdout", "String?", false, "Bounded stdout sample"), | ||
| ("stderr", "String?", false, "Bounded stderr sample"), | ||
| ("artifacts", "[String]", true, "Written artifact paths"), | ||
| ("screenshot", "HostSimulatorScreenshotMetadata?", false, "Simulator screenshot orientation and pixel metadata"), |
| ], | ||
| successShape: "{ ok, action, simulatorUDID?, apps[]?, app?, bundleID?, path?, target?, sourceCommand? } or { ok, action, plistPath, value?, preferences? } or { ok, action:app.prefs.set, plistPath, key, previousValue?, newValue, restartAdvice } or enhanced open-url { ok, status, hostAction, ready?, snapshot? }", | ||
| failureShape: "{ ok:false, error:{ code, message, hint, nextAction? } }", | ||
| outputSemantics: "Use app for host-side install, launch, terminate, open-url, container, and preferences. For smoke readiness, prefer open-url with --wait-ready --snapshot.", |
…6-sim-screenshot-orientation # Conflicts: # docs-linhay/memory/2026-06-04.md
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
screenshotmetadata toHostActionOutputfortriton sim screenshot --json.pixelWidth/pixelHeight.raw-simctl-framebufferwithnormalizationApplied=falseandnormalizationStrategy=metadata-only.host.simulator-screenshot-metadataso agents do not treat raw simctl framebuffer output as display-normalized evidence by default.Closes #26
Validation
Because the feature worktree directory is not named
TritonKit, SwiftPM local package identity resolves the root package as the worktree slug andCLI/Package.swiftcannot resolvepackage: "tritonkit". I validated via a temporary copy at/tmp/tritonkit-issue26-copy/TritonKitwithout changing package configuration.Result: all passed.
Remaining risk
This PR is intentionally metadata-only. It does not rotate or normalize the PNG yet; the output still preserves
xcrun simctl io screenshotraw framebuffer orientation. A future change can add normalization if TritonKit can reliably obtain the simulator display orientation.